-
-
Notifications
You must be signed in to change notification settings - Fork 226
Replace MauiTestUtils with DeviceRunners nuget package #3628
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ope_OnTask_PropagatedToCaller
bruno-garcia
approved these changes
Oct 3, 2024
Member
bruno-garcia
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a bit concerned with the Skip calls. otherwise LGTM
test/Sentry.Maui.Device.TestApp/Platforms/Android/SentryInstrumentation.cs
Outdated
Show resolved
Hide resolved
Member
|
Feel free to merge once u're comfortable with maintaining this, from my PoV looks good |
This was referenced Oct 6, 2024
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The device tests use a device runner Test App, which is a very simple MAUI application containing project references to all of our test projects. XHarness can be used to install and launch the test app on iOS and Android physical devices or simulators, run any xUnit tests referenced and then capture both test results and application logs as artefacts.
The mechanism used to discover and run xUnit tests within the test app is called Device Runners. Originally Microsoft made the Device Runners available as some sample code in the
dotnet/mauirepository, which we vendored into (i.e. copied into) thetest/MauiTestUtilsfolder of thesentry/sentry-dotnetrepo.Since then, that code appears to have been turned into a semi-official Device Runners NuGet package, which is being actively maintained.
This PR removes our vendored in code - using a package reference to the most recent version of the NuGet package instead.
Hopefully this will make it easier for us to upgrade the device tests to net8.0 in a separate PR.
Notes to Reviewers
It looks like a lot to review but mostly it's deleting code:
tests/MauiTestUtilsdirectory, all of which is being deleted, it starts to look less overwhelmingtest/Sentry.Maui.Device.TestApp/Platforms/Android/SentryInstrumentation.csclass is worth calling out... it's a custom instrumentation class that enables us to intercept launching of the device tests on Android (so we can inject environment variables that have been set on CI - mainly so we can skip certain flaky tests in CI)Notes on Flaky Tests
There are a few tests that I had to disable on iOS and some others I had to disable on Android. On Android, it was only when targeting API 31 or 33 (the only two I tried other than 27). When this happens, the error message from Xharness is a bit misleading - it indicates the tests are timing out after 900 seconds.
This may be related to Restrictions on non-SDK interfaces but, if so, I don't understand how.
You can download a CSV describing ~656k API interfaces and the availability of each... so it's possible some of the APIs we're using in our device tests are not supported on API 33. How to know whether we're using any of the blocked/obsolete APIs though??? Maybe that information is in the adb logs if you know how to look for it but our adb logs are about 33k lines so that's a bit like trying to finding a needle in a haystack.
If we could work out what was going on there, we might be able to re-enable the Android tests. I don't have any leads for the iOS tests.
#skip-changelog